1. Technical Field
The present invention relates to electronic camera apparatus comprising an electronic camera for developing an image signal of a viewed scene, whereon an indication of saliency of the viewed scene is developed.
2. Background Art
The electronic camera may be of the still or video variety, or any hybrid version. That is to say it may be adapted for example for providing one or more temporally spaced image frame signals in response to an input command (still camera), or for providing a continuous succession of image frame signals in response to an input command (video camera), or for providing a continuous sequence of image frame signals interspersed with still image signals (where the latter may have greater resolution if desired) (hybrid camera).
Whereas formerly a camera user might be expected to take a great deal of care in composing a picture before recording it at the optimum time, particularly where photographic film was involved, the advent of electronic cameras with digital image recording, and more recently rapid increase in available practical memory size, has increased the ease with which a large number of still images, or a long video sequence, can be recorded electronically. This is tending to lead to situations where a camera is left recording over an extended period of time, with the intention of subsequently sifting through the stored images to select only the more interesting images or sequences for further processing. Such further processing might involve permanent storage, copying, reproduction, transmission and/or printing for example. In addition, the stored images may need to be organised and/or indexed for easy retrieval.
Nevertheless, as the amount of stored image data increases, there comes a point at which the time required to analyse it all eventually outstrips the time and patience of the user. In such a case, the recorded images are then not used at all, so that potentially interesting images are lost, or all of the data must be viewed, so that the potentially interesting images are lost in a welter of uninteresting material. It can be seen that in either case the provision of too large a storage facility tends to be counterproductive.
The commonly assigned UK Patent No. 2380348, issued Oct. 1, 2003, entitled “Determination of Features of Interest by Analysing the Movement of Said Features Over a Plurality of Frames” relates to automated camera apparatus comprising an electronic camera, the camera comprising a photoelectric sensor optically coupled to imaging optics for providing a video signal representative of an image viewed by the camera, and securing means for securing at least the imaging optics on the head of a wearer with its axis generally parallel to the normal forward line of sight of the wearer, the apparatus further comprising video signal processing means arranged to receive said video signal for judging the degree of interest shown by the wearer in a feature in the scene and for providing a related saliency signal, the video signal processing means including image analysis means for identifying the presence of features and their locations relative to a reference position in the frame within each of a series of frames in a time window of the video signal, and movement analysis means coupled to the output of said image analysis means for analysing the movement of at least one said feature over said series of frames to provide said saliency signal.
This apparatus automatically derives a measure of saliency which is then used at the time or later for processing of the images or image signals from the camera. The saliency measure is derived from external cues such as changes in the gaze of the camera user.
Another form of apparatus is described by Andrea Lockerd in “LAFCam—Leveraging Affective Feed back Camcorder”, ACM CHI 2002 Conference on Human Factors in Computing Systems, Minneapolis, Minn., Apr. 20-25, 2002. A video camera user wears a head mounted camera turned to view the facial expression of the user, a glove for sensing skin conductivity and a head mounted microphone for detecting laughter. The latter two can be used as saliency indicators when reviewing or editing the resulting video, and the facial expression can be presented as a picture within the video image for the same purpose. However, each of these measures is principally that of an involuntary action, not under the control of the user.
Other patent disclosures where a saliency or related measure is automatically determined include European Patent Application No. EP 1 109 132 (Eastman Kodak); U.S. Pat. No. 5,956,026 (Sharp); and International Patent Application No. WO 00/79800 Telefonaktiebolaget L M Ericsson). A number of papers in this respect have also been published, examples being:—                1. “Summarizing Video Using a Shot Importance Measure and a Frame Packing Algorithm”, by Shingo Uchihashi et al, in Proc ICASSP 1999, vol 6, pp 3041-3044.        2. “Context Triggered Visual Episodic Memory Prosthesis 2, by Jyrki Hoisko, in Proc ISWC 2000, Atlanta Ga.        3. “Structuring Personal Activity Records Based on Attention-Analyzing Videos from Head Mounted Camera”, by Yuichi Nakamura et al, in Proc ICPR 2000, Barcelona.        4. “Video Retrieval and Relevance Feedback in the Context of a Post-Integration Model”, by Rucyu Roy Wang et al, in Workshop on Multi-Media Signal Processing (MMSP2001), 3rd to 5th Oct. 2001, Cannes, France.        
However, the most pertinent measure of saliency in many circumstances will be the amount of interest actually felt by the camera user, or the amount of importance actually attached to a picture by the user, and while external clues can give some indication of these factors, they cannot always be assessed accurately and automatically therefrom.
The present invention is concerned with camera apparatus comprising an electronic camera for producing an image signal, and a physically or mechanically operable user control for receiving an input from a user and generating a saliency signal while the image signal is being produced.
European Patent Application No. EP 0 860 980 describes a digital camera with the capability of allowing the user to examine pictures already stored, to decide various parameters, such as the number quality and size of any desired prints, which parameters are then stored for later use. However, as has already been discussed above, this means that the camera user has to review each of the images after they have been taken.
More pertinently, Australian Patent No. 743216 (Canon) discloses a video capture apparatus for capturing a series of video frames comprising a level of interest (LOI) input means operable by the user to generate a LOI signal which is recorded together with the video frames for later use in editing the recorded signals.
As in Canon, the present invention involves an immediate attribution of saliency while the pictures are being taken, and the saliency can be stored with the pictures, so as to reduce the amount of time and effort needed by the user to obtain the most desirable pictures during subsequent editing processes. In some respects, this may be viewed as a development of the conventional camera picture button, where the user operates the button once the required picture is in view, i.e. once it has been decided that the picture has sufficient saliency. Where the saliency signal is stored, this enables relatively easy selection of the requisite picture at a later time from a larger number of images. Furthermore, there is relatively little additional effort in obtaining a multi-valued saliency signal during picture taking, so that refinement of a picture selection process (for example) is possible with little or no additional work once the picture and saliency signals have been recorded.
However, as will be explained later, an immediate attribution of saliency is useful in other ways, such as managing the memory, and controlling the transmission and/or compression of image signals, which features are not available on conventional camera apparatus.
Many cameras incorporate both an on/off switch for energising the camera electronics prior to picture taking, and also a picture taking control. For example, a digital video camera may have an on/off switch which, inter alia, serves to enable the viewing screen, together with a separate picture taking trigger which is operated for taking a video clip. A still photographic film camera, on the other hand, may include a single switch button having a first position for energising the electronics including camera settings such as focus, timing and aperture, and a further position at which the shutter is triggered. In both cases however, there is a clear distinction between the initial energisation step and the control of the picture taking.
It might be considered that the mere operation of the camera button, trigger or other control for picture taking is an indication of saliency, particularly in situations where signal storage is limited. Commonly there is a resulting signal which is at least partly electrical, for example for operating the electronics of an electronic still or video camera to take and store image(s), or even for operating flash on an otherwise purely mechanical camera, and conventionally it will be binary in nature.
However, as mentioned above, it would be desirable to be able to switch a camera on for taking pictures “continuously”, for example all the time for video, or at regular periods for a still camera, regardless of the pictures it is taking, and to inspect or otherwise use the stored images at a later date. Under these circumstances, the fact of having the camera picture taking control actuated to provide a binary signal provides no indication whatsoever of the saliency of the stored images.